Wireless augmented reality communication system

ABSTRACT

The system of the present invention is a highly integrated radio communication system with a multimedia co-processor which allows true two-way multimedia (video, audio, data) access as well as real-time biomedical monitoring in a pager-sized portable access unit. The system is integrated in a network structure including one or more general purpose nodes for providing a wireless-to-wired interface. The network architecture allows video, audio and data (including biomedical data) streams to be connected directly to external users and devices. The portable access units may also be mated to various non-personal devices such as cameras or environmental sensors for providing a method for setting up wireless sensor nets from which reported data may be accessed through the portable access unit. The reported data may alternatively be automatically logged at a remote computer for access and viewing through a portable access unit, including the user&#39;s own.

RELATED APPLICATIONS

This application is based on provisional patent application Ser. No.60/115,993 filed Jan. 15, 1999.

GOVERNMENT LICENSE RIGHTS

The U.S. Government has certain rights in this invention pursuant toNAS7-1407 awarded by NASA.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention is a wireless augmented reality system (WARS) thatleverages communications and multimedia information processingmicroelectronics, along with displays, imaging sensors, biosensors, andvoice recognition to provide hands-free, tetherless, real-time accessand display of network resources, including video, audio and data.

2. Description of the Prior Art and Related Information Onlineinstruction manuals are becoming more prevalent in the industrial andeveryday environment. These electronic technical manuals (ETM) may beinteractive. Just as with printed manuals, ETMs may become verydifficult to use and maintain in these environments where elements of anenvironment, such as dust, chemical or general harshness may bedetrimental to the electronics and storage devices used to display andoperate the ETM. Further, it is not always possible for a worker whorequires access to an ETM to stop work to consult ETM.

These problems are multiplied in extraterrestrial environments such as aspace shuttle or a space station. During intra and extra vehicularactivities, it may be virtually impossible to access a traditionalkeyboard and computer display to access an ETM. For example, during asatellite repair mission, it would not be practical for an astronaut ina bulky extravehicular space suit to type commands on a keyboard to viewa display in the extreme environment of outer space where the sun glaremay make viewing impossible.

Hands-free portable computers have been implemented in an attempt tosolve some of these problems. For example, U.S. Pat. Nos. 5,305,244 and5,844,824 describe systems in which a head-up display and voicerecognition is implemented in a portable computer for displaying ETM.However, these systems, being a single user-to-computer paradigm, do notallow multiple-user access to multiple computers, multimedia devices ornodes on a network for accessing arbitrarily-selected data channels.Further, these previously-described systems are self contained and theirdata storage needs to be updated periodically to be sure that the latestdata is displayed. Further, these systems do not allow two-waycommunication over local and wide area networks to other multi-mediausers and devices, and do not provide real-time biomedical informationabout the physical condition of the user.

There is thus a need for a wireless, wearable communications systemallowing two-way voice, video and data communication between local usersand to remote users and devices over network nodes, along withtetherless real-time monitoring of the local user's physical condition.

SUMMARY OF THE INVENTION

The system solves the above problems with prior art systems with anadaptive wireless remote access network comprised of small individualportable access units linked to a local cellular general purpose node.Interlinked general purpose nodes support communications acrossdifferent habitat modules or internal-to-extravehicular communications,in the case of the space environment; terrestrial wired networks such asthe Internet can serve as the interconnection of remotely scatteredaccess nodes in an industrial, commercial or home environmentapplication.

The system may provide shuttle and international space stationastronauts with tetherless, on-demand access to data channels frommultimedia devices such as cameras or audio sensors associated withother persons or in a stand-alone configuration, and multimedia or datadisplay from a networked computer terminal and to the equipment controlcapabilities which may be available through that computer. Transparentto such access, the system can maintain a data channel for monitoring anastronaut's health or environment via in-situ sensors. Though thissystem may be used for the shuttle and the international space station,the system has uses in many possible applications related to medical,industrial, and commercial areas.

The invention is a personal communications system designed especiallyfor the space shuttle or station environment to provide cellularcommunications access throughout the vessel with video, audio, data andcomputer connect services. A small, wearable portable access unit (PAU)communicates over high-rate link to a centrally-located network accessunit, called a general purpose node herein. The system backbone provides2-way video, 2-way audio, and a multi-purpose data channel between thePAU and general purpose node. One embodiment of the PAU used forpersonal communication has an attached headset with video display, audiofeed and camera, which together may be used for audio or videoteleconferencing. When used as a virtual terminal to a computer in thenetwork, the user is able to view and manipulate imagery, text or video,using voice commands to control the terminal operations.

Using the system, an astronaut may efficiently operate and monitorcomputer-controllable activities inside or outside the vehicle orstation. Hands-free access to computer-based instruction texts, diagramsand checklists replaces juggling manuals and clipboards, and tetherlesscomputer system access allows free motion throughout a cabin whilemonitoring and operating equipment. Along with voice commands, anintegrated “touchpad” on the PAU may be used for remote computer controlthrough a sensor data channel; this return data channel may also be usedfor other control data as from a three-D mouse or data glove inputdevice, allowing the real-time video display to be used for remote,wireless monitor and control of robotic cameras or manipulators.

Concurrent with information provided to the astronaut, the system alsoallows external observation of the astronaut's situation; personalbiological or other sensors can send back continuous telemetry throughpersonal access unit and general purpose node. A miniature cameraintegrated into the headset provides real-time video of the wearer'sfield of view to remote observers. In this way, for example, a principalinvestigator located on Earth may consult with a payload specialist onthe operation or troubleshooting of their equipment.

The system provides wireless high-rate data exchange. The radio link isadapted to operate within a high-interference, high-multipathenvironment of a space shuttle or space station module. Radio frequency(RF) links do not require visual line-of-sight to operate, but the metalwalls and lack of RF absorbers, combined with moving human bodies,creates an enormous potential for destructive self-interference of theradio signals. The integrated radio and multimedia data processingtechnology provides for efficient and high-quality video and audio datacompression for noisy indoor communications channels. The systemsupports multiple-user access for video, audio, and sensor data servicesin the wireless coverage area. Potential applications of the system arein any environment where heads-up, hands-free information retrieval ormultimedia communications access improves efficiency includingtetherless operations/monitor consoles, remote consultations in medicalor maintenance procedures, and hazardous/confined space activities.There are also in-the-home entertainment/communications applications.

Similar to the space extravehicular activities applications,bio-isolation suits have similar operation constraints to space suits.They are worn commonly where there are chemical or biologicalcontaminates, and any extraneous materials brought into a chamber, suchas clipboards or documents, also present a contamination risk. A unitsuitably modified for use in a space suit could be used in thissituation. This allows the user to use a computer (log data, use a checklist, etc.), to communicate with colleagues, including providingfirst-hand video of work in progress, and to maintain constantmonitoring of the health of the user.

An extension of the medical applications areas would be in remotetelemedicine. Many medical diagnostic and treatment tools are being madeportable and rugged enough to be taken to remote sites. Some examplesare an ultrasound unit that is the size of a backpack, an entireintensive care unit of equipment built into a stretcher, and a traumapod built into a cruise missile. For many of these devices, CRT or LCDpanels comprise a significant amount of the bulk and weight of thedevices. The system of the present invention may provide a replacementfor the CRT or LCD panel as well as an interface to the control systemof the device, while providing communications access through aninterface to the remote site's existing communications equipment.

Industrial applications include use by inspection or maintenance crewsin remote or dangerous environments such as oil refineries, drillingrigs, power plants, etc., where the personnel can move around with theirhands and peripheral vision free to attend to their own safety andtasks. They would be in constant contact with the information theyneeded and any technical assist could be given by individuals looking atthe return video images from the user.

An example of a commercial application is for mission control and otheroperations personnel who presently must sit at a display console forhours at a time. These individuals could make use of the system of thepresent invention to increase their mobility and efficiency.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagrammatic illustration of the components of the system ofthe present invention;

FIG. 2 is block diagram illustrating communications components used bythe personal access unit and general purpose node of the system of FIG.1; and

FIG. 3 is a flowchart illustrating a method performed using the systemof FIG. 1.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

With reference to FIG. 1, a diagram illustrating components of thesystem of the present invention is shown. The system may comprise smallpager-like devices called portable access units 100. The portable accessunits 100 are accessorizable for different “multimedia” interfaces fordisplay, camera, audio and sensor operation. Another embodiment of theportable access unit 100 a comprises a wearable headset and microphoneassembly 102 a.

The portable access units 100-100 a interface directly through wirelesslink with a network through a general purpose node 150. The generalpurpose node 150 allows wireless-to-wire communication with a localnetwork 170. The local area network 170 may be electrically connected toa wide area network or Internet 172 in order to connect to remote localarea networks 174. Alternatively, the general purpose node 150 may bedirectly connected to the wide area network 172. The general purposenode 150 may thus act as a router for routing video, display, audio andcontrol data packets between the portable access units 100 and other, orremote, portable access units 100 or remote media devices 125, 180, etcconnected to the networks 170-174. The connection with a network 170-174may occur directly in electrical connection with one of the networks170-174, or in wireless communication through a remote general purposenode 150 a that is electrically connected to the network. The portableaccess units 100 may provide communication to and from remote mediadevices comprising computers 180-182 running specialized client softwareor certain commercial multimedia Internet software products such asvideo conferencing products that adhere to the industry standard H.323for multimedia data transfer.

Each portable access unit 100-100 a may dynamically associate with theclosest general purpose node 150-150 a when it is logged on to thenetwork 170-174 or is connected thereto. Each general purpose node150-150 a records the associations and registers each portable accessunit 100-100 a on a list of connections associated with the particulargeneral purpose node 150-150 a. The list of connections is stored in arandom access memory device included in the general purpose node 150-150a. When a portable access unit 100 is logged off or disconnected fromthe network 170-174, it is disassociated from the particular generalpurpose node 150-150 a that it was associated with, and thus, is removedfrom the list of connections.

As shown on an example selection list screen 190 that may be presentedon a display 102 or headset 102 a on any of the portable access units100-100 a, the user can set up a video, audio, or data link with anyother portable access unit 100-100 a or remote media device 125, 180,etc, that is logged onto a network 170-174. The headset 102 a maycomprise a heads-up display (120 in FIG. 2) inside a headset embodying atransparent color LCD device. Using control keys or voice commands, auser of the portable access unit 100-100 a may select a local or remoteportable access unit 100-100 a on a selection list 190 of other portableaccess units 100-100 a or media devices 125, 180. The selection list 190comprises a combination of the lists of connections stored on all of thegeneral purpose nodes 150-150 a. Users may further access a nameserverlocated on the access node 150 for locating remote unfamiliar portableaccess units 100-100 a or remote media devices.

By selecting entries from the selection list 190, users may communicatewith portable access units 100-100 a or various media devices such ascameras 125, internet phones 104, one or more computers 180-182 locatedthroughout the networks 170-174. A user may further select from the list190 user names representing users of other portable access units 100that are logged in and associated with remote general purpose nodes 150a connected to the networks 170-174.

With reference to FIG. 2, the components of the access node 150 and thewearable headset embodiment of the portable access unit 100 a is shown.Elements for both the general purpose access node and portable accessunit 100 a include a communications device 202. Data processingfunctions are implemented in the form of an audio/video coder/decoder(codec) pair 200, one codec 200 comprising part of the portable accessunit 100 a and the other codec 200 being part of another portable accessnode 100 a or remote media device for which it is desired to exchangesignals. At a portable access node, the codec 200 controls a digitaldata stream which is fed to the communications device 202, which isimplemented as an RF modem transceiver pair with an equivalentcommunications device 202 located in the general purpose access node.The codecs 200 serve as the interfaces to the external elements(including possibly the user display 102 a and the sensor 104) on bothsides of the communication continuum comprising the communicationsdevice 202 of the general purpose node 150, an internal networkinterface protocol circuit 152, the external networks 170-174 and theelectrical connection or general purpose access node connection to thedesired remote portable access node or media device. The internalnetwork interface protocol circuit 152 may comprise an Ethernet chip,memory and a network protocol chip. With this architecture, the systemaddresses the issues of multiple-access and data channel quality,through the implementation of the communications device 202. Multipleimplementations of the communication device 202 in the general purposenode 150 allows for multiple simultaneous communication links with aplurality of portable access units 100-100 a for the general purposenode 150.

With the base functionality of the communications device 202 and codecsubsystem 200, the architecture provides flexibility in utilization ofdifferent external components such as different headset 102 aconfigurations, sensor 104 packages, and network interface 152capabilities.

The communication device 202 is designed to operate in a high multipathspace station or terrestrial indoor environment while being able tosupport multiple users at high, multimedia-type bandwidths. Thus thecommunications device's 202 physical (PHY) and media access (MAC) layersin combination support multiple access, dynamic network association,channel error rates of broadcast video quality (1×10e-6), and data ratesup to broadcast-quality video bandwidths (on the order of 768 kbps peruser (one-way)). Modulation to achieve this performance will bedifferential phase-shift keying, of binary or higher order (quadratureor high-order quadrature amplitude modulation); the order chosenreflects the necessary user data volume to be supported within fixed,FCC-specified bandwidth allocations. Orthogonal frequency divisionmultiplexing, code division multiple access, and frequency hopping/timedivision multiple access may be used for achieving multiple access.Spread spectrum, channel equalization, antenna diversity andretransmission techniques may also be used for improving the reliabilityof the communications link. Through a combination of these technologies,two-way multimedia channel throughputs can be achieved for each ofmultiple simultaneous users. A variety of RF frequencies may be used,but the determining factor in frequency band selection is theavailability in the band of a relatively large amount of spectrum in thespace station or FCC terrestrial allocations, allowing the transmissionof compressed video. Ranges in the 2.5 to 5.7 band range are preferabledue to the FCC bandwidth available, the compactness of RF elementsrequired at these frequencies, and the potentially low amount ofinterference that will be sustained. The RF front end of both theportable access unit 100-100 a and general purpose node 150-150 a may beinterchangeable with different frequency front ends for system use indifferent frequency bands.

Low-rate, single user implementations of the communications system maybe effected through adapted commercial wireless-LAN type productsfollowing the FCC 802.11 standard such as a frequency-hopping 2.4 GHzwireless LAN transceiver by Waveaccess, Inc of Wellesley, Mass., ordirect-sequence spread-spectrum 2.4 GHz wireless LAN chipset by HarrisPrism of Melbourne, Fla. These radio implementations, as with commercialimplementations of the industry-proposed Bluetooth and HomeRF standards,will be limited in user access and overall throughput, however, andtherefore unsuitable to real-time video teleconferencing for multipleusers. The preferred embodiment for full capability is to implement thecommunications devices' physical and media access control layers incustom ASIC circuits allowing for support of all system capabilitieswithin microelectronics architecture for small size and low power draw,providing pager-type form factor of wearable personal access units100-100 a.

The communications device 202 comprises a buffer memory and a radiofrequency front end. Data modulation/demodulation circuits and errordetection/correction protocol circuits are further included. Variouscombinations of these circuits may be obtained from Proxim of Sunnyvale,Calif., Harris of Melbourne, Fla. and Stanford Telecom of Stanford,Calif. Alternatively, all of the various circuitry may be implementedwith an application specific integrated circuit (ASIC), or a combinationof an ASIC and discrete elements for size and weight efficiency.

Three classes of headsets 102 a may be used: hi-resolution militarysystems which are CRT based and may be provided by Honeywell ofMorristown, N.J., or Hughes Network Systems of San Diego, Calif.; mediumresolution industrial systems which are CRT or LED based scanners andmay be provided by Intervision of Santa Clara, Calif.; or low to mediumresolution entertainment systems which are color viewfinder LCD basedsystems that may be supplied by Virtual Vision of Redmond, Wash. (theV-CAP and E-GLASS), Sony Europe of Hampshire, United Kingdom (GLASSTRONVISOR) or Olympus of San Jose, Calif. Typical headset display 120specifications for the portable access unit 100 a include the following:

RESOLUTION: Comparable at least to VGA (640×480) or better to 1280×1024w/off-the-shelf display & I/O configuration

DISPLAY: >10 FL/day, Display Bright.Ratio: >2, Brightness range: 2OOM_(max)

FOV: 40-60 deg, Gray scale: >12

EyeRelief: 20-26 mm TSP, 14/10 mm (on/off-axis) exit pupil

Unif: 2:1 across 2/3 FOV, GLARE: <2.5% image content, PixelContrast: 25

FOCUS: Hands off, Obs: % look-around, Diopter range: ±2,

Mag: 1±p5%, Cont: >95%, motion sensor 10° cone, Inter. Eye. adj: 52-72mm

Image Enhan & IFF: Weaponsight, motion sensor and computer interface

The audio/video codec 200 in a portable access unit 100-100 a or otherclient device is based around a single chip, standards-based codec thataccepts analog or digital audio and video (i.e. NTSC or VGA) compressesthis input, and multiplexes the compressed data with an external datastream. The preferred industry standards are: ITU H.263 based video, ITUG.722 based audio, and ITU H.221 based multiplexing. The audio videocodec 200 in the portable access unit 100-100 a can establish a linkwith a similar audio/video codec 200 associated with another portableaccess unit 100-100 a or a remote media device 104, 125, 180 or 182. Thesignals from the codec 200 in the portable access unit 100 a outputs thereceived and decompressed remote signals from the device for which thelink was established. The interface between the codec 200 andcommunication device 202 as well as between the communication devices202 of the general purpose node 150-150 a and portable access unit100-100 a operate two-way with a high bandwidth suitable fortransmitting video. Of this bandwith, the audio portion utilizes up to64 kbps and the data from the sensor 104 utilizes the required amountfor the type of sensor 104, with the remainder allocated to compressedvideo. The quality of the video at these data rates in excess of 128kbps is at least equivalent to video teleconferencing quality video.

The audio/video codec 200 portion of the portable access unit 100-100 amay further comprise video input and output ports, audio input andoutput ports, data input and output ports, and a the above-mentionedmultimedia processor chip for packaging signals for data compression anddecompression for transmission. Exemplary multimedia processors includethe VCPEX chip by 8×8, Inc. of Santa Clara, Calif. or digital signalprocessing chips by Texas Instruments and others. The audio/video codec200 further comprises a field processor gate array, electricallyprogrammable read-only memory and random access memory for processingand packaging signals for compression and decompression

The sensor 104 may comprise a commercially available pulse oximetersensor or other type of bio-sensor. A pulse-oximeter sensor allows themeasurement of pulse rate and oxygen saturation of the blood. Data fromthe sensor 104 is transmitted to the general purpose node 150-150 a, andtransmitted to any remote media device connected to any of the networks170-172. The sensor 104 may comprise an “on body” wireless humanperformance and fatigue monitoring system that communicates with abelt-mounted transceiver/control module. The remote media device maycomprise a processor 180-182 for display or logging of the real-timesensor signals.

The headset 102 a comprises a heads-up display 120 comprising atransparent color LCD device for video signals received and processed bythe codec 200. The headset 102 a may further comprise, or have attachedthereto, an integrated microphone 122 for receiving voice commands fromthe user of the portable access unit 100 a or for communicating voicesignals to a remote portable access unit 100 or remote media device. Theheadset may further comprise a speaker 124 or earpiece unit forpresenting audio signals to the user. The portable access unit 100 a mayfurther comprise a digital camera 106 that may either be attached on theuser's person, or to the headset 102 a for providing video signals toother portable access units 100-100 a or media devices.

With reference to FIG. 3, a flow diagram illustrating the methodperformed by the system of FIG. 1 is shown. A user puts on the headset102 a, portable access unit 100 a, step 400. The user may log into thelocal general purpose node 150 wherein the portable access unitassociates with the general purpose node 150 such that the user is addedto a connection list stored in a random access memory device residing inthe general purpose node 150, step 401. Data is provided from thegeneral purpose node 150 to the portable access unit through thecommunication devices 202, step 402. The user is presented with aselection list 190 of portable access units 100-100 a and media deviceslogged onto the system on the display 120, step 404. The user selectsone of the entries from the selection list, step 406. The selection istransmitted to the general purpose node 150, step 408. The generalpurpose node 150 sets up a connection over the networks 170-174 forchanneling data between the portable access unit 100 a and the selectednetwork device, step 410. The selected network device may comprise theprocessor 180 or other network client 182 for running a softwareapplication, a camera 125 for providing remote viewing operations to theuser on the display 120, the Internet phone 104 for providing voicecommunications with the a remote user, or another portable access unit100-100 a over a remote general purpose node 150 a. By providing controlcommands to the microphone 122 or other input system, such as a keyboardor handheld mouse, the user may conduct operations by transmittingcommands between the portable access unit 100 a and the general purposenode 150 which routs the control commands to the device that the userselected, step 412.

It will thus be seen that changes may be made in carrying out the abovesystem and method and in the construction set forth without departingfrom the spirit and scope of the invention, it is intended that any andall matter contained in the above description and shown in theaccompanying drawings shall be interpreted as illustrative and not in alimiting sense.

1-33. (canceled)
 34. A method for providing wireless augmented realitythrough a network, the method comprising: wirelessly accessing thenetwork from a portable access unit through a general purpose nodecoupled in communication with the network, the general purpose node alsocoupled in communication with a processor device; associating theportable access unit with the processor device; capturing video of asubject matter in the portable access unit; providing a signal from theportable access unit to the processor device; responsive to providingthe signal, receiving information related to the subject matter; anddisplaying the information on the portable access unit.
 35. The methodof claim 34, wherein displaying the information comprises displaying theinformation as an overlay to the subject matter.
 36. The method of claim34, wherein displaying information comprises displaying the informationin a head set as an overlay to the subject matter.
 37. The method ofclaim 34, further comprising: sending commands to the processor devicefrom the portable access unit.
 38. The method of claim 34, furthercomprising: presenting a list of remote portable access units availablefor connection; and establishing two-way communications with a remoteaccess unit through the general purpose node.
 39. The method of claim34, wherein receiving information comprises receiving video information.40. The method of claim 34, wherein receiving information comprisesreceiving audio information.
 41. The method of claim 34, whereinreceiving information comprises receiving text information.
 42. Aportable access unit for receiving information from a processor devicethrough a network, the method comprising: a communication device towirelessly access the network through a general purpose node coupled incommunication with the network, the general purpose node also coupled incommunication with the processor device, the general purpose node toassociate the portable access unit with the processor device; a camera,coupled to the communication device, the camera to capture video of asubject matter in the portable access unit; and a display device,coupled to the communication device, the display device to outputinformation related the subject matter, the information received fromthe processor device responsive to providing the signal.
 43. Theportable access unit of claim 42, wherein the portable access unit is ahand-held device.
 44. The portable access unit of claim 42, wherein thedisplay device displays the information as a heads up display.
 45. Theportable access unit of claim 42, wherein the display device displaysthe information as an overlay to the subject matter.
 46. The portableaccess unit of claim 42, wherein the display device comprises a headset.
 47. The portable access unit of claim 42, further comprising: acodec, coupled to the communication device, to the camera, and to thedisplay device, the codec to compress video received from the camera andto decompress video received from the processor device.
 48. The portableaccess unit of claim 42, further comprising: a microphone, coupled tothe communication device, the microphone to receive audio commands forthe processor device.
 49. The portable access unit of claim 42, whereinthe communication device comprises an IEEE 802.11-compatiblecommunication device.
 50. The portable access unit of claim 42, whereinthe information comprises video information.
 51. The portable accessunit of claim 42, wherein the information comprises audio information.52. The portable access unit of claim 42, wherein the informationcomprises text information.
 53. A method for providing wirelessaugmented reality through a network, the method comprising: wirelesslyaccessing a network from a portable access unit through a generalpurpose node coupled in communication with the network, the generalpurpose node also coupled in communication with a plurality of processordevices; presenting the plurality of processor devices available forconnection; associating the portable access unit with a processor devicefrom the plurality of processor devices; capturing video of a subjectmatter in the portable access unit; providing a signal from the portableaccess unit to the processor device; responsive to providing the signal,receiving information related to the subject matter; and displaying theinformation through as a heads up display overlaying the informationrelative to the subject matter.